I semi-regularly receive emails with attachments from a certain sender. I very much want these emails, I have paid money for the attachments.

Most of the time but not always, GMail puts these emails directly into trash, bypassing the Inbox. That's Trash, not Spam.

Last week I received two of these emails within minutes of each other. One went to the Inbox, the other into Trash.

The sender is in my contact list.

The instructions to Whitelist an email address seem only to be designed to prevent the message going to Spam. These messages are going to Trash. I do have about a dozen Spam filters that send certain incoming emails directly to Trash, but none of those filters should have any effect on these emails that I want to keep. I am filtering --> Trash the following words: sexually, penis, sex, pills, viagra, vigara, medications, blonde, drugs, pharmacy, casino, promotion, T-Mobile, Samsung, and raffle.

Is it possible that the Spam filter is searching for those words inside the attachments? If so, that would explain a lot... (The attachments are zipped eBooks.)

What should I do to ensure that these emails get to my Inbox?

tanstaafl.

I have no direct answer to your question, but...

Do you really need those manual spam filters? I use Gmail for all my email and I almost never receive spam (say: maybe one email every three months). If not, I would simply remove those manual spam filters and rely on the standard Google spam filter. For me it works great!

The simplest experiment is to get rid of your hand-crafted anti-spam filters. Gmail's spam filtering is pretty good these days, and the spammers have so many different ways of misspelling Viagra that you'll never catch them all.

Pharma spam, in particular, has been under vigorous attack from the academic community and law enforcement. There's a lot less of it now than there used to be.

Doug, I've tried to emphasize before that creating your own spam filters is NOT the best way to fight spam in GMail. I'm not surprised that you're seeing this behavior. I'm sure that it's one of your filters that's causing the problem.

I think that whether or not GMail scans for those terms in attachments will depend on how you've created the filter. GMail is capable of searching attachments when you do simple mail searches, for example, so it's certainly capable of it, though it probably also depends on what file format those ebooks are.

I strongly recommend that you remove all those filters and let GMail do its thing. It's going to scan for spam anyway, so just spend a couple weeks marking spam using the built in tool and I promise that after a while you won't see any spam. I haven't seen any in years. The only thing I get is the occasional well-worded phishing scam, but it's tough for GMail to find those unless enough people mark it as phishing.

If GMail is filtering the attachments in addition to the body text, then it's not one of the filters... it's probably just about all of them causing the problem.

I will get rid of the filters and see if the problem goes away. If it does, that will be an indication that GMail is filtering the attachments.

I presently get one or two messages in Spam per day; but I see a lot of messages that my filters have kicked directly into Trash. Now they will end up in Spam where I'll have to deal with them. (I keep my Spam folder empty.)

I'll just have to see how it goes.

tanstaafl.

That's the root of your problem.

You should just remove all of your trash filters, let Gmail deal with your spam, and ignore the contents of your spam folder from now on.

The reason you see all those messages kicking into the trash folder is that it's processing your filters both on inbox messages and on spam messages. Just because one of your filters kicked something into the trash, that doesn't mean Google's spam filter didn't catch it. Google's spam filter probably did catch it, but your filter put it into the trash anyway.

The problem with you using filters to search for spam is, you personally can't possibly ever put in enough rules to correctly deal with spam. Google has spent much more time and development money on their spam-filtering engine, and it's the best. It beats every other method for spam filtering that I've ever seen. There's no way a simple word-matching filter will be satisfactory for you... As you've already seen, you're getting bitten by the false positives. Which is the main problem at the heart of any spam filtering engine.

Nothing that you do in your own personal filters is going to have a better false-positive/false-negative record than what Google already provides.

Doug,

I'm with everyone else I don't think your wanted emails will be marked as spam if you turn off your rules. I use Gmail for my personal account. Definitely for the first few days and maybe a week or two it had a number of false positives but now they are few and far between and false negatives are very rare too.

I really don't think you can expect absolute 100% spam detection performance via Google or your rules or anything short of a human. But I would say Google heuristics probably beats yours.

If anything make sure your wanted emails are not marked as spam via rules rather than the other way around.

Well, the [empirical] evidence seems to suggest that GMail IS filtering on the attachments as well as the title, address, and body of the email itself.

Since deleting all of my filters, my spam (in the Spam folder) has gone up by more than 10x, but my "problem" account with the eBook attachments has stayed in the Inbox where it belongs.

Who woulda thunk it!

tanstaafl.

The "spam" isn't really going up. It's just going into a different folder. Previously your rules were just putting that spam in a different place (the Bin/Trash) and also you were getting false positives from your rules.

Keep an eye on the spam folder every now and then. It does sometimes get it wrong (although if your rules hadn't done anything then it would still have marked it as spam).

Oh, I do. It's just now I empty five or six emails out of it three or four times a day instead of emptying maybe two emails out of it once a day. Yes, I know I'm anal about keeping my Spam and Trash folders cleaned out, but it makes me happy.

tanstaafl.

My spam folder says it has 1264 items

I never touch my spam folder; it currently has 2,201 items in it. Since it deletes spam after 30 days, this works out at ~75 pieces per day

Oh man I need to get more spam

You can have all of mine.

6467. Do I win?

In my experience (an N of two, my wife and me), if you actually take the time to empty the spam folder, you get less and less.

My wife never looks into her spam folder, and gets 20 or 30 a day. I empty mine regularly (I have a few lists that occasionally wind up there, and need to return those messages to the fold). I typically get 5 or maybe 10 a day. Before I began actively emptying it, I was getting 40 or 50 a day.

I suspect after you've assertively deleted certain types of spam, it gets more fully blacklisted, and gmail no longer delivers it to your spam folder, even.

As far as I can tell, Google is opaque on the issue.

-jk

I believe you are mistaken about this.

My reasoning: If that were true, then the end-user would have no recourse in the case of false-positives. I don't believe that any company implementing a spam filter, no matter how good that filter is, would trust the filter so much as to leave the user no recourse.

The difference in the amount of spam between your wife and yourself is likely due to other factors. It's normal for two people to get different amounts of spam.

I think you are both wrong.

Every email provider of any scale discards somemuch likely spam before it gets anywhere near their users. They use blacklists and honey pots to reject email from IP addresses or blocks of IP addresses that they suspect of sending spam. When they reject like this they don't even wait for the body of the email to rejected, they just refuse the message completely.

But also, I doubt Google take note of what spam you delete. Deleting spam from the spam folder doesn't indicate any reliable intent. You could just as easily be looking in the spam, reading a message you realise isn't spam but then still just delete it when you were finished.

My significant reduction in spam coincided with my beginning to delete it actively. Again, very small N. Perhaps my deleting a message aggregates in the decision to entirely block a sender. Who knows.

I've looked to see if I could discover how Google treats spam, but it seems an opaque process - entirely sensible if they don't want to arm the enemy.

-jk

Andy is pretty much right on, as I have to still ensure my own private mail server does similar. By pure numbers, about half of the messages that hit my server are blocked before they hit the actual spam filter that looks at content. Any of those messages are silently rejected and users have no idea they were targeted.

There are also spikes in activity, tied to the rise and fall of botnets out there and hacks against databases to extract addresses.

For example, since 4am this morning (so about a 10 hour window of activity):

112 attempts to deliver a message were stopped before accepting the message, with 101 of these being due to blacklists. The remaining were 8 that contained incorrect addresses or attempts to use my server as a relay, and 3 were manual e-mail address blocks added by my users. The last 3 are handy, as my users can use throwaway addresses and later blacklist them.

138 messages made it past the blacklist or address validation, meaning the server accepted the full content. 137 were sent to users inboxes, 1 was sent to a users spam folder.

I kept a very close eye on the system when it was set up, sharing Tony's view on false positives initially. With the numbers I see, it's not worth turning up the noise to accept the one or two false positives that might happen each year. In the rare occasion I've seen a false positive, it's usually been because the senders mail server was misconfigured and flagged as a spam relay. The admins of these servers can fix their issue and have them removed from blacklists in a quick and efficient way. Every reject will have a log message the sender can see in their mail server logs. It includes a URL to see why they were blocked and how to remove themselves.

I did avoid implementing one of the old spam blocking methods where a server built it's own whitelist over time. Main reason being that they would do this by sending back an automated e-mail upon first sighting of a new from address. Thats just creating more traffic for no good reason.

You probably won't be surprised to hear I still run my own mail server

(though I'd switch all my users to Google if there was a cost effective way of doing it)

That still could have been pure coincidence. Possible reasons:

1. Improvements in the blocking features of Gmail. As pointed out by others, some messages just get blocked at the gateway, never even getting the chance to hit the heuristic spam filter. It's possible that Gmail (or someone else whose systems are part of the relay chain) is making constant improvements to this, so the number of messages in your spam folder is naturally decreasing over time.

2. There are occasional news reports of certain spam-related resources being taken down. For instance, maybe a particular spam botnet gets ferreted out and eradicated. Maybe a particular open relay gets shut down. Maybe a particular company or individual who specializes in sending spam gets their financial channel cut off and they stop their activity. This kind of thing happens all the time, and when it does, we frequently see news reports of how there was a sudden drop in spam frequency on that day.

I would be very surprised to see that Gmail is watching what you delete and modifying their spam filters as a result. I think it even less likely that they are somehow creating a special heuristic spam filter specific to your account.

There's one exception to my statement, though, and that's this:

If, when you delete the message, there is some other option being made available, something to the effect of "always delete all messages from this sender", and you're selecting that option, then it would be automatically creating a filter for you, and might explain the behavior you saw. However, when I delete messages from my Gmail spam folder, I don't see an option like this available. So I'm doubtful that's what's happening in your case.

In the past several weeks, the ONLY spam I have received has been from gambling sites, at the rate of about 20 per day. All have similar or identical subject lines, but different sent-from addresses. It is surprising to me that GMail doesn't filter these out as a category, but I confess I have no idea how they would do that.

tanstaafl.

Was that Spam in your inbox, or spam directed by google to your spam folder?

Spam folder.

I almost never get spam in my inbox, maybe a couple times a year is all. As everybody has said, Google does a great job of filtering out spam. Since I am a bit anal about my email, I check and empty my spam folder several times a day, just on the off chance that something in there warrants my attention. This has happened maybe twice in the five years or so I have been using Gmail.

Presently my inbox has three items in it requiring future attention. My Spam and Trash folders are empty. And my saved emails are all nicely organized.

tanstaafl.